Method and apparatus for image capturing and simultaneous depth extraction

ABSTRACT

A system for image capturing and depth extraction includes a camera and a data processor. The camera includes: a spectrum coded aperture including at least two regions that pass spectrum channels of an incident light field which are different from each other; and a sensor configured to record the at least two spectrum channels to form an image captured in a sensor basis. The data processor is configured to convert the image captured in the sensor basis into an image of a processing basis, extract a disparity from the image of the processing basis, and convert the disparity into depth information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Russian Patent Application No.2014127469, filed on Jul. 4, 2014 in the Russian Patent Office andKorean Patent Application No. 10-2015-0083666, filed on Jun. 12, 2015 inthe Korean Intellectual Property Office, the disclosures of which areincorporated herein in their entirety by reference.

BACKGROUND

1. Field

Apparatuses and methods consistent with exemplary embodiments relate tocomputational photography, and more particularly, to light fieldcapturing and processing.

2. Description of the Related Art

One of the main applications of light field photography is in extractionof image depth information. Examples of apparatuses for light fieldcapturing or image depth information extraction may include a stereocamera, a plenoptic camera, a camera with a binary coded aperture, and acamera with a color coded aperture. However, these apparatuses mayrequire additional space, increase costs of cameras, or cause areduction in optical efficiency.

SUMMARY

Exemplary embodiments address at least the above problems and/ordisadvantages and other disadvantages not described above. Also, theexemplary embodiments are not required to overcome the disadvantagesdescribed above, and may not overcome any of the problems describedabove.

One or more exemplary embodiments provide methods and apparatuses forlight field capturing and processing by using information in an cameraand a data processor.

According to an aspect of an exemplary embodiment, there is provided asystem for image capturing and depth extraction including: a lenssystem; a spectrum coded aperture including at least two regions thatpass spectrum channels of an incident light field which are differentfrom each other; and a sensor configured to record the at least twospectrum channels to form an image captured in a sensor basis; and adata processor configured to convert the image captured in the sensorbasis into an image of a processing basis, extract a disparity from theimage of the processing basis, and convert the disparity into depthinformation.

The different spectrum channels may form a basis of the spectrum codedaperture.

The processing basis may be different from the sensor basis and thebasis of the spectrum coded aperture.

The spectrum coded aperture may have three regions, and the threeregions may include a transparent region in a central portion, and tworegions having spectrum bandwidths respectively corresponding to yellowand cyan.

The processing basis may three vectors, and the three vectors mayinclude a vector corresponding to yellow, a vector corresponding tocyan, and a vector perpendicular to the two vector.

The spectrum coded aperture may include two regions having spectrumbandwidths respectively corresponding to yellow and cyan.

The processing basis may include three vectors, and the three vectorsmay include a vector respectively corresponding to yellow, a vectorcorresponding to cyan, and a vector perpendicular to the two vector.

The spectrum coded aperture may include three congruent regions havingspectrum bandwidths respectively corresponding to yellow, cyan, andmagenta.

The processing basis may include vectors corresponding to yellow, cyan,and magenta.

The spectrum coded aperture may include three non-congruent regionshaving spectrum bandwidths respectively corresponding to yellow, cyan,and magenta.

The processing basis may include vectors respectively corresponding toyellow, cyan, and magenta.

The spectrum coded aperture may have a smooth bandwidth change over anaperture region.

The spectrum coded aperture may be fixed to the lens system.

The spectrum coded aperture may be attachable to and detachable from thelens system.

The spectrum coded aperture may be moved from an optical train that doesnot participate in the image formation.

The captured image may be an image selected from a video sequence.

The spectrum coded aperture may insert the image selected from the videosequence into the lens system.

The spectrum coded aperture may be inserted into an aperture stop of thelens system.

The lens system may include a single lens and the spectrum codedaperture may be located in the lens.

The spectrum coded aperture may correct a previous video image of thevideo sequence acquired by the sensor.

The spectrum coded aperture may have a combination of an opaque regionand a congruent region, and the congruent region may be transparent ortransmit ultraviolet light, infrared light, or visible light.

The spectrum coded aperture may have a combination of an opaque regionand a non-congruent region, and the non-congruent region may betransparent or transmits ultraviolet light, infrared light, or visiblelight.

The spectrum coded aperture may be a spatial light modulator (SLM).

The data processor may include a preprocessing unit configured toperform the converting the captured image, a disparity estimation unitconfigured to perform the extracting the disparity, and a conversionunit configured to perform the converting the disparity to the depthinformation.

The data processor may further include an image restoration unitconfigured to restore the captured image based on the extracteddisparity.

According to another aspect of an exemplary embodiment, there isprovided a method of image capturing and depth extraction including:recording at least two shifted spectrum channels of a light field toform an image captured from a video; converting the captured image intoan image of a processing basis; estimating a disparity based on acorrelation between pixels of the spectrum channels in the processingbasis to extract a disparity map; restoring the captured image based onthe extracted disparity map; and converting the disparity map into adepth map.

The estimating of the disparity may include: generating candidate imageshaving respective shifts in the spectrum channels; computing matchingcost involved in the candidate images in the spectrum channels;propagating a matching cost involved in a low textured region of thecandidate images; and estimating a matching cost having a sub-pixelaccuracy based on the propagated matching cost.

The correlation between the pixels of the spectrum channel forrequesting the disparity estimation may include a correlation metriccomputed in a sparse moving window.

The correlation between the pixels of the spectrum channel forrequesting the disparity estimation may be computed by using at leastone stereo matching algorithm.

The computing of the correlation by using the stereo matching algorithmmay include sum of absolute differences (SAD), normalized crosscorrelation (NCC), or Laplacian image contrast (LIC).

The correlation metric may include a fast Fourier transform (FFT).

The correlation metric may include a recursive exponential filter (REF).

The restoring of the captured image may include performing imageblurring.

The restoring of the captured image may include performing a spectrumchannel alignment in the processing basis.

According to another aspect of an exemplary embodiment, there isprovided a mobile device for image capturing and depth extraction inultraviolet light, infrared light, or visible light including: a lenssystem; at least one spectrum coded aperture including at least tworegions that pass spectrum channels of an incident light field which aredifferent from each other; a sensor configured to record the at leasttwo spectrum channels to form an image captured in a sensor basis; and acoded aperture fixture configured to move at least one spectrum codedaperture relatively with respect to the lens system; and a dataprocessor configured to convert the image captured in the sensor basisinto an image of a processing basis, extract a disparity from the imageof the processing basis, and convert the disparity into depthinformation.

The coded aperture fixture may be configured to replace at least twospectrum coded apertures in an optical train.

The coded aperture fixture may be configured to shift all the spectrumcoded apertures from the optical train.

The coded aperture fixture may be inserted into an aperture stop.

The spectrum coded aperture may have a combination of an opaque regionand a congruent region, and the congruent region may be transparent ortransmit ultraviolet light, infrared light, or visible light.

The spectrum coded aperture may have a combination of an opaque regionand a non-congruent region, and the non-congruent region may betransparent or transmits ultraviolet light, infrared light, or visiblelight.

According to another aspect of an exemplary embodiment, there isprovided an apparatus for image capturing including: a lens system; atleast two spectrum coded apertures including a first aperture and asecond aperture which have different characteristics of opticalefficiency and depth discrimination from each other; a coded aperturefixture adapted to dispose the first aperture in front of the lenssystem; and a data processor configured to obtain depth information ofan image captured through the first spectrum coded aperture, and controlthe coded aperture fixture to determine whether to switch the firstaperture to the second aperture based on the depth information.

The first aperture may include a transparent region placed in the centerof the first aperture and two regions separated by the transparentregion. The two regions pass different color spectrums, respectively.

The two regions may pass a yellow spectrum and a cyan spectrum,respectively.

The second aperture may include equally divided two regions which maypass yellow and cyan spectrums, respectively.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and/or other aspects will be more apparent by describingcertain exemplary embodiments, with reference to the accompanyingdrawings, in which:

FIG. 1 is a diagram of a depth extraction/image restoration apparatusaccording to an exemplary embodiment;

FIGS. 2A to 2F are diagrams of spectrum coded apertures according toexemplary embodiments;

FIGS. 3A to 3I are diagrams for describing a channel shift;

FIG. 4 is a high-level outline diagram of a depth informationextraction/image restoration method according to an exemplaryembodiment;

FIG. 5 is a diagram for describing a parabola fitting according to anexemplary embodiment; and

FIGS. 6A to 6D are diagrams for describing a depth extraction/imagerestoration apparatus according to an exemplary embodiment.

DETAILED DESCRIPTION

Exemplary embodiments are described in greater detail below withreference to the accompanying drawings.

In the following description, like drawing reference numerals are usedfor like elements, even in different drawings. The matters defined inthe description, such as detailed construction and elements, areprovided to assist in a comprehensive understanding of the exemplaryembodiments. However, it is apparent that the exemplary embodiments canbe practiced without those specifically defined matters. Also,well-known functions or constructions are not described in detail sincethey would obscure the description with unnecessary detail.

As used herein, the term “and/or” includes any and all combinations ofone or more of the associated listed items.

It will be understood that when a region is referred to as being“connected to” or “coupled to” another region, it may be directlyconnected or coupled to the other region or intervening regions may bepresent. It will be understood that terms such as “comprise”, “include”,and “have”, when used herein, specify the presence of stated elements,but do not preclude the presence or addition of one or more otherelements.

FIG. 1 is a diagram of a depth extraction/image restoration apparatus101 according to an exemplary embodiment. The depth extraction/imagerestoration apparatus 101 may include a camera 102 and a data processor103. The camera 102 may include optical lens (objective lens) 104, aspectrum coded aperture 105, and a sensor 106. The spectrum codedaperture 105 may be inserted into an optical system which is constitutedby the combination of the lens 104, the sensor 106, and other opticalparts. The spectrum coded aperture 105 may be placed in an optical paththat a ray of light follows through the optical system. The spectrumcoded aperture 105 may be a diaphragm plane. The sensor 106 may beconfigured to discriminate different spectrum bandwidths from oneanother. For example, the sensor 106 may be a sensor covered with amosaic color/spectrum filter array, or a color stacked photodiodesensor. The data processor 103 may include a preprocessing unit 108, adisparity estimation unit 109, an image restoration unit 110, and adisparity-to-depth conversion unit 111. The data processor 103 mayreceive a raw image 107 captured by the camera 102. The preprocessingunit 108 may convert the captured image 107 from a sensor basis to aprocessing basis in which a spectrum coded aperture filter may not bepresent. The disparity estimation unit 109 may perform disparityestimation. Then image restoration unit 110 may perform imagerestoration. The disparity-to-depth conversion unit 111 may performdisparity-to-depth conversion on optical system parameters.

The spectrum coded aperture 105 may be divided into sub-regions thatrespectively have spectrum passbands. The number, geometric structures,and spectrum passbands of the sub-regions may be changed according toapplications of optical efficiency, a depth map, and color imagerestoration image quality. Some of them are illustrated in FIGS. 2A to2F.

FIGS. 2A to 2F are diagrams illustrating patterns of various spectrumcoded apertures having a tradeoff relationship among the opticalefficiency, the depth map, and the color image restoration imagequality. For light field coding, spectrum filters f₁, f₂, and f₃ may beused. Examples of the spectrum filters f₁, f₂, and f₃ may include avisibly recognizable color filter, an infrared/ultraviolet filter, and amulti-path filter having two or more passbands

Main characteristics of a spectrum coded aperture are opticalefficiency, depth discrimination ability, and color image restorationimage quality. The highest depth discrimination index may be obtainedfrom a geometric structure of a spectrum coded aperture having thelongest distance between the centers of aperture sub-regionscorresponding to respective optical spectrum bands. FIG. 2A shows anaperture pattern that has a relatively long distance between the centersof sub-regions f₁, f₂, and f₃ and a relatively small filter size in thesub-regions. Consequently, an opaque region of the coded aperture may beincreased so that the optical system has a reduced optical efficiency.If the aperture design is deformed to enhance optical efficiency asshown in FIG. 2B, the typically extracted disparity accuracy may bedeteriorated.

For specific applications, there may exist a tradeoff between opticalefficiency and depth discrimination ability. For example, FIG. 2C showsa geometric structure of an aperture having a cyan filter f₁ (i.e.f₁=f_(Cyan)) and a yellow filter f₂ (i.e., f₂=f_(Yellow)) on halves andFIG. 2D shows a geometric structure of an aperture having a transparentsub-region f₂ a cyan filter f₁ (i.e., f₁=f_(Cyan)), a yellow filter f₃f₁ (i.e., f₃=f_(Yellow)), and a green filter f₄ (i.e., f₄=f_(Green)).Here, the yellow filter may have a passband including green and redlight spectrums. The cyan filter may have a passband including green andblue light spectrums. The transparent region may not filter incominglight. The green channel may not be distorted by these filters and maybe used as a reference in an image restoration process. In comparisonwith the aperture structure of FIG. 2D, the aperture structure in FIG.2C may have a better depth map. However, the aperture structure of FIG.2D may have an superior optical efficiency to the aperture structure ofFIG. 2C. FIG. 2A shows an aperture having a circular filter and anopaque region, which may be used to obtain a high-quality depth mapimage when light is excessive. The aperture structure of FIG. 2A maycompensate for excessive light directed to the camera 102. An aperturestructure having infrared light and ultraviolet light on halves as shownin FIG. 2C may be a fully opened aperture and may have the same opticalefficiency and have excellent potential with respect to depthextraction. However, an additional process such as image restoration andphotograph array correction may be performed for an image capturedthrough the aperture structure of FIG. 2. FIG. 2E shows a spectrum codedaperture having three or more spectrum sub-regions with a hivearrangement and FIG. 2F illustrates a spectrum coded aperture having asmooth bandwidth change over an aperture region.

The light field, which is corrected by the spectrum coded aperture 105,may be input to the image sensor 106 that generates the captured rawimage 107.

The light field having passed through the spectrum coded aperture 105may be coded. That is, the light field may be divided into differentspectrum parts by passing through corresponding aperture sub-regions.Therefore, different views may be extracted from a single captured imagewith respect to the same scene by dividing the single captured imageinto spectrum channels correspondingly with respect to the spectrumcoded aperture.

FIG. 3A illustrates the captured image 107 obtained by a sensor 106 thatis capable of discriminating the corresponding spectrum bandwidth withrespect to the spectrum coded aperture described above with reference toFIG. 2B. In the optical system, a position of a defocused object 302 inFIG. 3A), which is obtained by the presence of the spectrum codedaperture, may be changed with respect to relatively correspondingspectrum filter positions as shown in FIGS. 3D, 3E, and 3F as comparedto a focused object 301 in FIG. 3A). Such a view may be used forextracting a disparity map and restoring the captured image 107. Theresults of image deblurring with respect to the spectrum channels areillustrated in FIGS. 3G, 3H, and 3I. A deblurred color image isillustrated in FIG. 3B. A deblurred image (restored image) aligned inthe spectrum channel is illustrated in FIG. 3C.

FIG. 4 is a high-level outline diagram of the data processor 103. Asystem input may be the raw image 107 captured by the camera 102. Inoperation 108, the captured image {Is₁, Is₂, . . . } 107 may bepreprocessed by denoising and demosaic technologies and be translatedfrom a sensor spectrum basis to a processing basis. In general, theprocessing basis may not be a spectrum filter. Is_(i) is an image colorchannel acquired by an optical system sensor. In order to perform such aconversion, a conversion matrix Π needs to be preferentially estimated.For simplicity, it is assumed that the camera 102 uses the aperturestructure having a cyan filter f₁ and a yellow f₂ as described abovewith reference to FIG. 2C, and a red, green, blue (RGB) mosaic colorfilter array.

w_(Cyan) and w_(Yellow) are color filters that represent cyan and yellowfilters in an RGB color space. In order to construct a conversion matrixthat has an excellent condition number and is capable of anon-degenerate inverse conversion, a third basis vector w_(X) is definedas a vector product w_(Cyan)×w_(Yellow). Vectors e_(r), e_(g), and e_(b)are respectively a red basis, a green basis, and a blue basis for thecamera sensor 106. In the sensor spectrum basis,w _(fi)=(p _(1i) e _(r) ,p _(2i) e _(g) ,p _(3i) e _(b))^(T) ,i=1,2,3

An auxiliary matrix Π is represented as follows:

$\Pi = {\left( {w_{Cyan},w_{X},w_{Yellow}} \right) = {\begin{bmatrix}p_{11} & p_{12} & p_{13} \\p_{21} & p_{22} & p_{23} \\p_{31} & p_{32} & p_{33}\end{bmatrix}.}}$

If the matrix Π is used, any observed color w may be decomposed by anaperture filter response.w _(filter)=Π⁻¹ w,

w_(filter) means a channel intensity in the spectrum filter basis (cyan,X, and yellow). The matrix Π may be inversely converted.{If_(Cyan),If_(X),If_(Yellow)} represents an image channel acquired inthe processing basis. In the case of a different number of basis vectorsin the sensor basis and the processing basis, an inverse conversionmatrix (a left inverse matrix and a right inverse matrix) may be used.

In operation 109, a disparity disp(i,j) may be estimated with respect toall pixels of the image. disp(i,j) is a matching cost for disparityestimation and may use a conventional cross-correlation method of ashifted spectrum channel corr(If_(Cyan) ^(d),If_(Yellow) ^(d)).

${{disp}\left( {i,j} \right)} = {\underset{d}{argmax}\left\lbrack {{corr}\left( {{{If}_{Cyan}^{d}\left( {i,j} \right)},{{If}_{Yellow}^{d}\left( {i,j} \right)}} \right)} \right\rbrack}^{2}$

A generalized mutual correlation metric may be used in the disparityestimation unit 109 so as to process an arbitrary number of spectrumchannels. {I_(i)}₁ ^(n) represents a set of nth acquired views in thenth acquired spectrum channel with respect to the same scene fromslightly different viewpoints. I_(i) represents an M×N frame. Aconventional correlation matrix M_(d) may be expressed by the set{I_(i)}₁ ^(n) and a disparity value d.

$M_{d} = \begin{pmatrix}1 & \ldots & {{corr}\left( {I_{1}^{d},I_{n}^{d}} \right)} \\\vdots & \ddots & \vdots \\{{corr}\left( {I_{n}^{d},I_{1}^{d}} \right)} & \ldots & 1\end{pmatrix}$where (*)^(d) means a parallel shift in a corresponding channel.

A determinant of the matrix M_(d) is a good measure of the mutualcorrelation {I_(i)}₁ ^(n). In practice, in a case where all channels arecompletely correlated, the matrix M_(d) is a singular matrix and thedeterminant thereof is 0. In another aspect, in a case where data iscompletely uncorrelated, the determinant of the matrix M_(d) is 1. Inorder to estimate the depth map by using such an operator, the disparityvalue d corresponding to the least value of the determinant det(M_(d))needs to be found from each pixel of the image.

Other operators for cost computation matching may be used. Examples ofthe operators may include conventional stereo matching metrics,Laplacian contrast metrics, and feature based metrics.

All statistic computations may use a conventional local moving window.However, in an exemplary embodiment, an exponential moving window may beused because this complies with a naturally sparse gradient prior andpropagates a matching cost with respect to a low textured region.Furthermore, an exponential kernel filtering may be efficiently computedby using a recursive 0(1) convolution in a spectrum domain.S _(n) =I _(n)·(1−α)+S _(n-1)·α,where S is a result of convolution with respect to an image I at an nthpixel, and a is defined as follows:α=e ^(−σ) ^(spatial)where σ_(spatial) is an exponential dampling factor that represents animage similarity required in a spatial domain.

This equation may also be used for computing an effective approximatevalue of a joint bilateral filter for propagating disparity informationon a small texture region.S _(n)=Disp_(n)·(1−α(n))+S _(n-1)·α(n),where Disp_(n) is a disparity of an nth pixel, and α(n) is a functionrepresenting the degree of similarity of an image color.α(n)=e ^(−σ) ^(spatial) ·e ⁻ ^(range) ^(·Δ(I) ^(n) ^(,I) ^(n-1) ⁾,where Δ(I_(n), I_(n-1)) represents the degree of similarity betweencolor images in a range domain.

Sub-pixel estimation may be performed by using a parabola fittingalgorithm as shown in FIG. 5. In parabola fitting, three given points,d_(k), d_(k−1), d_(k+1) may be taken into consideration. d_(k) may berepresented as argmax_(d) det(M_(d)) (i.e., d_(k)=argmax_(d)det(M_(d))), and d_(k−1) and d_(k+1) may be set as a previous argumentand a next argument, respectively. A variable of a maximum value of aunique parabola satisfying {d_(k−1),det(M_(d) _(k−1) )},{d_(k),det(M_(d) _(k) )}, and {d_(k),det(M_(d) _(k) )} may beanalytically computed in the following formula.

$d_{\max}^{{sub}\text{-}{pixel}} = {d_{k} - \frac{b}{2a}}$where α=0.5(d_(k+1)+d_(k−1))−d_(k) and b=0.5(d_(k+1)−d_(k−1))

The image restoration unit 110 may perform preliminary image restorationIr(x,y) based on the disparity estimation. The captured image of FIG. 3Amay be deblurred as shown in FIG. 3B. A color alignment of the deblurredimage may be performed as shown in FIG. 3C. FIG. 3A illustrates anexample of the image captured by the system. FIG. 2B illustrates ageometric structure of a spectrum coded aperture. The system may befocused on one object 301 and another object 302 may be defocused. Thedefocused object 302 captured by the camera 102 may cause a spectrumchannel misalignment in a photo array to the extent that the blurredimages 305, 306, and 307 as shown in FIG. 3D, FIG. 3E, and FIG. 3F areblurred with respect to a conventional imaging system. The imagedeblurring may be performed based on a deconvolution technology and beapplied to images corresponding to different disparity values. Forexample, while the focused object 301 does not require the deblurring,the images 305, 306, and 307 of the defocused object 302 in therespective spectrum channels are deblurred with respect to the disparitylevels thereof. The deblurred image of FIG. 3B is still misaligned withrespect to the spectrum channels f₁, f₂, and f₃, as shown in FIGS. 3G,3H, and 3I. Misalignment vectors {right arrow over (s₁)}, {right arrowover (s₂)}, and {right arrow over (s₃)} respectively corresponding tothe spectrum channels f₁, f₂, and f₃ may be estimated at the respectivepositions of the captured image 302. A restored image

304 may be acquired by the aligned spectrum channel, based on themisalignment vectors {right arrow over (s₁)}, {right arrow over (s₂)},and {right arrow over (s₃)}.

_(ι)(x,y)=If _(i)(x+s _(ix) ,y+s _(iy)),where i is the number of spectrum channels, and s_(ix) and s_(iy) areprojections in an x-axis direction and a y-axis direction of a vector{right arrow over (s_(ι))}, respectively.

The image may be converted from a spectrum filter basis {If₁, If₂, . . .} to a device play unit basis {I₁, I₂, . . . }. The imaging system has avignetting effect that results in a reduction of an image's brightnessat the periphery of the image, as compared to the center of image. Insuch a system, the vignetting effect may be mathematically alleviated bythe following equation.I _(i,j) ^(restored) =U _(i,j) ·I _(i,j),where I_(i,j) and I_(i,j) ^(restored) are a captured image and arestored image at an (i,j) pixel, respectively. U_(i,j) is anunvignetting coefficient previously computed once during the calibrationof the optical system.

${U_{i,j} = \frac{I_{i,j}^{ideal}}{I_{i,j}}},$where I_(i,j) and I_(i,j) ^(ideal) are a captured image and anunvignetted image of a known image at an (i,j) pixel, respectively.

In a case where the coded aperture is present, the unvignettingcoefficient U_(i,j) needs to be independently computed with respect toeach spectrum channel. This process may be performed by the imagerestoration unit 110.

A final image refinement process may be used to reduce artifact causedby inaccurate disparity estimation. Technologies based on a human'svisual perception (for example, bilateral filtering, median filtering,or the like) and natural image priors (for example, sparse gradientprior, color lines prior, or the like) may be used.

The placement-to-depth conversion unit 111 may convert the disparitydisp(i,j) into a depth map d(i,j) 114 with respect to a single lensoptical system by using generalized optical system parameters 112generalized in a thin lens formula.

${{\frac{1}{z_{1}} + \frac{1 + {d\text{/}D}}{z_{2}}} = \frac{1}{f}},$where f is a lens center distance, and z₁ and z₂ are distances from eachlens to an object plane and an image plane, respectively.

This formula for a complex object may depend on the design of theoptical system.

The above-described image capturing apparatus may be extended forperforming a temporal coding and a spectral coding. The temporal codingmay be performed while moving the spectrum coded aperture with respectto the image capturing apparatus. This extension may remove a motionblur as well as a known defocus blur caused by a movement of thespectrum coded aperture.

The above-described image capturing apparatus may extract depthinformation from a photograph as well as a video stream that isappropriately encrypted by the coded aperture and is appropriatelyregistered by a detector array. In addition, the spectrum coded aperturemay be modified so as to mix a photograph and depth information on theimage captured according to the presence or absence of the spectrumcoded aperture. For example, the depth map extraction process may beperformed by just using a key frame (for example, every Nth frames) of avideo sequence, and other frames may be restored by using imageinformation and a depth map of the key frame. This process may increasetime efficiency and image quality of the system.

Furthermore, the type of the spectrum coded aperture and the geometricstructure may be changed according to the image automatically capturedby the detector array. For example, when light is excessive, theaperture including the circular filter and the opaque region, asillustrated in FIG. 2A, may be used instead of reducing the exposuretime or increasing the f-number of the optical system.

The depth extraction/image restoration apparatus according to theexemplary embodiment may be included in mobile phone camera or webcamera equipment, but is not limited thereto. The depth extraction/imagerestoration apparatus according to the exemplary embodiment may be usedin a compact optical camera.

FIG. 6A is a diagram of a permanently fixed color coded aperture in anoptical system of a camera, according to an exemplary embodiment. Sincelight passes through a fixed color filter aperture, the image quality ofa color image may degrade. Each color band may be projected at differentpositions of a photograph array causing a ghost image effect. A depthestimation and a color image restoration may be performed by theabove-described depth estimation method.

FIG. 6B is a diagram of a color coded aperture in which an opticalsystem is movable by a mechanical or electromagnetic unit, according toan exemplary embodiment. In a three-dimensional (3D) mode, the colorcoded aperture may be present in an optical system to acquire depthinformation on a scene and a computatively restored color image. In atwo-dimensional (2D) mode, the color coded aperture may not be presentin an optical system that captures an original 2D image withoutdistortion.

As shown in FIG. 6B, at least two spectrum coded apertures may beattached to the smartphone. The slider (also referred to as an aperturefixture) may switch between the spectrum coded apertures, for example,according to a control signal from the data processor 103. However, thepresent embodiment is not limited thereto, and the spectrum codedapertures may be switched manually or under the control of a centralprocessing unit (CPU) in the smartphone. When an image is capturedthrough one of the spectrum coded apertures, the data processor 103 mayextract depth information from the captured image and determine whetherto change the aperture to another one based on the depth information.For example, if the data processor 103 determines that the depthdiscrimination of the image does not meet a requirement preset by a userinput, the data processor 103 may send a control signal to the slider sothat the previously used aperture is changed to another one which isknown to have a better depth discrimination ability.

FIG. 6C is a diagram of a spectrum coded aperture with a spatial lightmodulator (SLM) capable of changing a spectrum passband of a coded coloraperture, based on time, according to an exemplary embodiment. Theapparatus of FIG. 6C may operate in a 2D or 3D mode as described abovewith reference to the exemplary embodiment of FIG. 6B.

In addition, the apparatuses of FIGS. 6B and 6C may also acquirealternating video frames. By changing the aperture before the frame isrecorded, one frame may be obtained in the 2D mode and another frame maybe obtained in the 3D mode. Consequently, the system may acquire twovideo streams. One video frame may include an original color frameacquired in the 2D mode, and another video stream may include a framesuitable for the depth extraction.

FIG. 6D is a diagram of a spectrum coded aperture that is attachable toa smartphone lens, according to an exemplary embodiment. Due to a largersize of an optical system, the apparatus of FIG. 6D may obtain moreexcellent depth map image quality as well as more excellent opticalefficiency and video image quality than apparatuses with the attachedspectrum coded aperture.

The apparatus according to the exemplary embodiment includes a spectrumfiltered aperture, and at least one of a RGB color filter, a red, green,blue, and white (RGBW) color filter, a cyan, magenta, yellow (CMY)filter, a cyan, magenta, yellow, green (CMYG) color filter, and aninfrared (IR) filter, but is not limited thereto. A combination ofsensors having color/spectrum spaces may be used.

The exemplary embodiment may be applied to any digital cameras,including a mobile phone camera, so as to perform mirror hardwaremodification and generate the disparity/depth maps having low costalgorithms. The acquired disparity map may be used in image splitting,custom blur type (bokeh), computational viewpoint disparity, imagefiltering, and digital post-refocusing having other special effects.

In addition, the term “unit” as used herein may mean a hardwarecomponent, such as a processor or a circuit, and/or a software componentthat is executed by a hardware component such as a processor.

While not restricted thereto, an exemplary embodiment can be embodied ascomputer-readable code on a computer-readable recording medium. Thecomputer-readable recording medium is any data storage device that canstore data that can be thereafter read by a computer system. Examples ofthe computer-readable recording medium include read-only memory (ROM),random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, andoptical data storage devices. The computer-readable recording medium canalso be distributed over network-coupled computer systems so that thecomputer-readable code is stored and executed in a distributed fashion.Also, an exemplary embodiment may be written as a computer programtransmitted over a computer-readable transmission medium, such as acarrier wave, and received and implemented in general-use orspecial-purpose digital computers that execute the programs. Moreover,it is understood that in exemplary embodiments, one or more units of theabove-described apparatuses and devices can include circuitry, aprocessor, a microprocessor, etc., and may execute a computer programstored in a computer-readable medium.

The foregoing exemplary embodiments are merely exemplary and are not tobe construed as limiting. The present teaching can be readily applied toother types of apparatuses. Also, the description of the exemplaryembodiments is intended to be illustrative, and not to limit the scopeof the claims, and many alternatives, modifications, and variations willbe apparent to those skilled in the art.

What is claimed is:
 1. A system for image capturing and depthextraction, the system comprising: a lens system; a spectrum codedaperture including at least two regions that pass spectrum channels ofan incident light field which are different from each other; and asensor configured to record the at least two spectrum channels to forman image captured in a sensor basis; and a data processor configured toconvert the image captured in the sensor basis into an image of aprocessing basis, extract a disparity from the image of the processingbasis, and convert the disparity into depth information.
 2. The systemof claim 1, wherein the different spectrum channels form a basis of thespectrum coded aperture.
 3. The system of claim 2, wherein theprocessing basis is different from the sensor basis and the basis of thespectrum coded aperture.
 4. The system of claim 1, wherein the spectrumcoded aperture has three regions including a transparent region in acentral portion, and two regions having spectrum bandwidths respectivelycorresponding to yellow and cyan.
 5. The system of claim 1, wherein theat least two regions of the spectrum coded aperture have spectrumbandwidths respectively corresponding to yellow and cyan.
 6. The systemof claim 1, wherein the spectrum coded aperture includes three congruentregions having spectrum bandwidths respectively corresponding to yellow,cyan, and magenta.
 7. The system of claim 1, wherein the spectrum codedaperture includes three non-congruent regions having spectrum bandwidthsrespectively corresponding to yellow, cyan, and magenta.
 8. The systemof claim 1, wherein the spectrum coded aperture has a smooth bandwidthchange over an aperture region.
 9. The system of claim 1, wherein thespectrum coded aperture is fixed to the lens system.
 10. The system ofclaim 1, wherein the spectrum coded aperture is attachable to anddetachable from the lens system.
 11. The system of claim 1, wherein thespectrum coded aperture has a combination of an opaque region and acongruent region, and the congruent region is transparent or transmitsultraviolet light, infrared light, or visible light.
 12. The system ofclaim 1, wherein the spectrum coded aperture has a combination of anopaque region and a non-congruent region, and the non-congruent regionis transparent or transmits ultraviolet light, infrared light, orvisible light.
 13. The system of claim 1, wherein the data processorcomprises a preprocessing unit configured to perform the converting thecaptured image, a disparity estimation unit configured to perform theextracting the disparity, and a conversion unit configured to performthe converting the disparity to the depth information.
 14. The system ofclaim 13, wherein the data processor further comprises an imagerestoration unit configured to restore the captured image based on theextracted disparity.
 15. A method of image capturing and depthextraction, the method comprising: recording at least two shiftedspectrum channels of a light field to form an image captured from avideo; converting the captured image into an image of a processingbasis; estimating a disparity based on a correlation between pixels ofthe spectrum channels in the processing basis to extract a disparitymap; restoring the captured image based on the extracted disparity map;and converting the disparity map into a depth map.
 16. The method ofclaim 15, wherein the estimating the disparity comprises: generatingcandidate images having respective shifts in the spectrum channels;computing a matching cost involved in the candidate images in thespectrum channels; propagating a matching cost involved in a lowtextured region of the candidate images; and estimating a matching costhaving a sub-pixel accuracy based on the propagated matching cost. 17.The method of claim 15, wherein the correlation between the pixels ofthe spectrum channel for requesting the disparity estimation includes acorrelation metric computed in a sparse moving window.
 18. The method ofclaim 15, wherein the correlation between the pixels of the spectrumchannel for requesting the disparity estimation is computed by using atleast one stereo matching algorithm.
 19. The method of claim 15, whereinthe restoring the captured image comprises performing a spectrum channelalignment in the processing basis.
 20. A mobile device for imagecapturing and depth extraction in ultraviolet light, infrared light, orvisible light, the mobile device comprising: a lens system; at least onespectrum coded aperture including at least two regions that passspectrum channels of an incident light field which are different fromeach other; a sensor configured to record the at least two spectrumchannels to form an image captured in a sensor basis; and a codedaperture fixture configured to move at least one spectrum coded aperturerelatively with respect to the lens system; and a data processorconfigured to convert the image captured in the sensor basis into animage of a processing basis, extract a disparity from the image of theprocessing basis, and convert the disparity into depth information. 21.The mobile device of claim 20, wherein the different spectrum channelsform a basis of the spectrum coded aperture.
 22. The mobile device ofclaim 20, wherein the processing basis is different from the sensorbasis and the basis of the spectrum coded aperture.
 23. The mobiledevice of claim 20, wherein the spectrum coded aperture has threeregions including a transparent region in a central portion, and tworegions having spectrum bandwidths respectively corresponding to yellowand cyan.
 24. The mobile device of claim 20, wherein the at least tworegions of the spectrum coded aperture have spectrum bandwidthsrespectively corresponding to yellow and cyan.
 25. The mobile device ofclaim 20, wherein the spectrum coded aperture includes three congruentregions having spectrum bandwidths respectively corresponding to yellow,cyan, and magenta.
 26. The mobile device of claim 20, wherein thespectrum coded aperture includes three non-congruent regions havingspectrum bandwidths respectively corresponding to yellow, cyan, andmagenta.
 27. The mobile device of claim 20, wherein the spectrum codedaperture has a smooth bandwidth change over an aperture region.
 28. Themobile device of claim 20, wherein the spectrum coded aperture is fixedto the lens system.
 29. The mobile device of claim 20, wherein thespectrum coded aperture is attachable to and detachable from the lenssystem.
 30. The mobile device of claim 20, wherein the spectrum codedaperture has a combination of an opaque region and a congruent region,and the congruent region is transparent or transmits ultraviolet light,infrared light, or visible light.
 31. The mobile device of claim 20,wherein the spectrum coded aperture has a combination of an opaqueregion and a non-congruent region, and the non-congruent region istransparent or transmits ultraviolet light, infrared light, or visiblelight.
 32. The mobile device of claim 20, wherein the data processorcomprises a preprocessing unit configured to perform the converting thecaptured image, a disparity estimation unit configured to perform theextracting the disparity, and a conversion unit configured to performthe converting the disparity to the depth information.
 33. The mobiledevice of claim 32, wherein the data processor further comprises animage restoration unit configured to restore the captured image based onthe extracted disparity.